The present invention benefits from priority of U.S. application Ser. No. 61/657,664, entitled “Real Time Denoising of Video,” filed Jun. 8, 2012 and U.S. application Ser. No. 61/662,065, also entitled “Real Time Denoising of Video,” filed Jun. 20, 2012. The contents of both documents are incorporated herein by reference.
The present disclosure relates to noise reduction in video data and, in particular, to real-time noise reduction techniques for such video data.
Many consumer electronic devices have camera systems that capture video data locally for storage or for delivery to other devices. The designs of the electronic devices may vary but, generally, the devices will include central processing units (“CPUs”) and graphical processing units (“GPUs”), memory systems, and programming constructs, such as, operating systems and applications that manage the device's operation.
A camera system generally includes an image sensor and an image signal processor. The image sensor may generate an output video signal from incident light. The image sensor's output may include a noise component that can be considered to be white (no frequency dependence) with a signal-dependent variance due to shot noise. It is largely un-correlated between color component channels (Red, Green, Blue). The image signal processor may apply various processing operations to the video from the image sensor, including noise reduction, demosaicing, white balancing, filtering, and color enhancement. At the conclusion of such processes, the noise components of the video signal are no longer white. Instead, the video noise may depend on the video signal, its frequency, illuminant, and light level, and also may be correlated between channels.
The problem of correlated noise is very significant in consumer electronic devices that have small sensors. The problem may not be as acute in digital single-lens reflex (“DSLR”) camera sensors where pixels may be fairly large. The problem may become particularly difficult, however, in consumer electronics devices for which the camera is merely a part of the system as a whole—laptop computer, tablet computers, smartphones, gaming systems and the like—where the sensors typically are less expensive and have smaller photodetector area to capture incident light. These sensors tend to have lower electron-well capacity, further deteriorating the signal-to-noise ratio (“SNR”)—especially in low-light situations.
Compounding the problem, the camera pipeline introduces a number of artifacts such as false edges, sprinkles, and black/white pixel clumps that, from a signal point of view, are not noise (actually they appear more like structures). These artifacts severely degrade image quality in low light.
Although such noise effects might be mitigated by increasing exposure time, doing so introduces other artifacts such as motion blur.
Although some spatial denoising solutions have been proposed, the complexity of many such operations render them inappropriate for real time processing of video data (e.g., high definition video at 30 frames per second) by CPU- and/or GPU-based software systems.
Accordingly, the inventors perceive a need in the art for video enhancement processing techniques that improve perceptual quality of video data with limited processing complexity to be amenable to real time processing of video by software.